The present disclosure relates to testing coverage in general, and to functional coverage, in particular.
Computerized devices control almost every aspect of our life—from writing documents to controlling traffic lights. However, computerized devices are bug-prone, and thus require a testing phase in which the bugs should be discovered. The testing phase is considered one of the most difficult tasks in designing a computerized device. The cost of not discovering a bug may be enormous, as the consequences of the bug may be disastrous. For example, a bug may cause the injury of a person relying on the designated behavior of the computerized device. Additionally, a bug in hardware or firmware may be expensive to fix, as patching it requires call-back of the computerized device. Hence, many developers of computerized devices invest a substantial portion of the development cycle to discover erroneous behaviors of the computerized device.
During the testing phase, a sample of all possible behaviors of the computerized device is inspected. Coverage tools for checking software provide a measure of how well the software being evaluated has been exercised during testing and thereby give a level of assurance that the software is of high quality.
There are a number of types of coverage known in the art, such as statement coverage, line coverage, condition coverage, path coverage, method coverage, and the like. One additional coverage method is functional coverage. Functional coverage is designed to measure amount, portion or a similar metric of tests that examined predetermined functional behaviors. Once functional coverage is measured, quality assurance (QA) personnel may design additional tests to examine untested behaviors.
Traditionally, the functional coverage model is defined by a QA staff member, an engineer, a programmer or the like. The functional coverage model comprises a set of attributes, each having a corresponding set of possible values. The functional coverage may define a set of coverage tasks, each comprising a combination of values of the different attributes. In some cases, each pair, triplet, quadruplet and the like of values of two attributes defines a coverage task.
Functional coverage checks coverage of coverage tasks in a trace. The trace comprises one or more entries, each entry indicative of a coverage task that was covered by a target device, system, program or the like (referred to hereinbelow as a target system). For example, each entry may describe a state of a transaction performed by the target system. The trace may be generated during execution of the target system. The entry may comprise values of functional coverage attributes. In some cases, some entries may not indicate a value of each attribute, but only for a portion.
The functional coverage model may further comprise a set of restrictions defining a series of values of different attributes that may not appear together. For example, consider a functional coverage defining two attributes: ACTION and USER. The ACTION attribute may be each of the following values: RETRIEVE, STORE, and MODIFY PERMISSION. The USER attribute may be each of the following values: ADMIN, USER, GUEST. In some cases, a guest user cannot modify permission. A restriction may be defined to indicate that the couple (GUEST, MODIFY PERMISSION) is not a valid couple. The fact that a trace does not comprise an entry covering a coverage task that includes the couple does not affect the functional coverage. In other words, all possible coverage tasks—which together form the maximal possible coverage in respect to a functional coverage—do not include any coverage task that comprises the restricted couple.